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Abstract 


Capturing hyperspectral images requires expensive and 
specialized hardware that is not readily accessible to most 
users. Digital cameras, on the other hand, are significantly 
cheaper in comparison and can be easily purchased and 
used. In this paper, we present a framework for reconstruct- 
ing hyperspectral images by using multiple consumer-level 
digital cameras. Our approach works by exploiting the dif- 
ferent spectral sensitivities of different camera sensors. In 
particular, due to the differences in spectral sensitivities of 
the cameras, different cameras yield different RGB mea- 
surements for the same spectral signal. We introduce an 
algorithm that is able to combine and convert these differ- 
ent RGB measurements into a single hyperspectral image 
for both indoor and outdoor scenes. This camera-based ap- 
proach allows hyperspectral imaging at a fraction of the 
cost of most existing hyperspectral hardware. We validate 
the accuracy of our reconstruction against ground truth hy- 
perspectral images (using both synthetic and real cases) 
and show its usage on relighting applications. 


1. Introduction 


Color is the visual perception or interpretation of light. 
Light is a continuous electromagnetic radiation over a range 
of spectrum (visible light ranges from 400nm to 700nm). 
The human vision system, as well as most cameras, sense 
this physical light through a tri-stimulus mechanism where 
three channels respond differently to the incoming light as 
follows: 


De = / o(A)eg(A)AaA, (1) 


2 


where p; is the output of the k*” channel, © is the range 
of the visible spectrum, o is the incoming light, and c;, 
represents the spectral response of the k“” sensor channel. 
For the vast majority of cameras, these three channels have 
spectral sensitivity that fall into the red, green, and blue 
ranges of the visible spectrum. 

While this three channel tri-stimulus representation is 
good for representing perceived color, it falls short of ex- 
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Figure 1. This image shows an overview of our system. We re- 
construct hyperspectral images by capturing images of a scene 
with multiple consumer cameras. Our system exploits the different 
spectral sensitivities of different cameras and convert their differ- 
ent color measurements into hyperspectral signals. 


plaining the full physical nature of light. For example, when 
different cameras are used, the same light spectral power 
distribution may result in different colors due to the dif- 
ferent spectral responses c;, of the cameras. In addition, 
two distinct spectral power distributions may result in the 
same R, G, B values on the same camera due to projection 
of the light onto only three color channels. Hyperspectral 
imaging (HSD), on the other hand, records a more accurate 
representation of physical light as it captures dense spectral 
samples across the visible wave lengths. The difference be- 
tween multispectral imaging and hyperspectral imaging is 
the number of bands captured. Multispectral imaging gen- 
erally captures a small number of bands (3 to 10 channels), 
while hyperspectral imaging usually records higher num- 
ber of channels. We refer to our approach as hyperspectral 
imaging as our goal is to sample the visible spectrum with 
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31 channels (every 10nm between 400nm and 700nm). 


Due to the physical nature of hyperspectral data, HSI 
has been effectively used for different applications that re- 
quire accurate measurements of light. For example, HSI has 
been used for cultural heritage analysis to record the spec- 
tral data of historical documents and paintings [10, 19, 29]. 
HSI has also been widely used for scientific applications 
such as earth science and remote sensing [9, 23], astron- 
omy [22], medical science, food science [24, 27] and com- 
puter vision [30]. 


The most significant drawback for working with hyper- 
spectral imaging is obtaining access to a hardware that is 
able to densely sample the visible spectra. Hyperspectral 
imaging devices typically have costs in the range of tens 
of thousands of dollars. Not surprisingly, only a hand- 
ful of researchers have access to such equipment. This is 
evident in the small number of datasets that are currently 
available [5, 14, 35]. There has been recent work that has 
exploited active illumination to build HSI systems [6, 31]. 
These methods multiplex varying illumination into a scene 
to recover the hyperspectral reflectance of objects. While 
such methods are more affordable, this type of HSI system 
requires a significant amount of expertise to build the neces- 
sary illumination infrastructure. In addition, such systems 
cannot be used outdoors as they rely on controlling the illu- 
mination in the scene. 


Contribution In this paper, we propose a novel algorithm 
to reconstruct a hyperspectral image of a scene from multi- 
ple images taken by different consumer cameras (Fig. 1). In 
particular, we propose an algorithm that uses the different 
spectral sensitivities of the different cameras to reconstruct 
the hyperspectral signal at different scene points. We cast 
this as an optimization problem that simultaneously esti- 
mates a bilinear system that models the spectral reflectance 
of scene points as well as the illumination spectrum. Our 
work leverages priors on the space of camera spectral sen- 
sitivities as well as the space of real world material and illu- 
mination. We describe an effective alternating-optimization 
framework that can solve this bilinear system and produce 
a high-quality hyperspectral image for both indoor and out- 
door scenes. This overall framework and corresponding op- 
timization algorithm enables an affordable and easy to use 
system for hyperspectral imaging. 


The remainder of this paper is organized as follows: Sec- 
tion 2 describes related work; Section 3 provides the details 
of our HSI framework including the problem formulation, 
analysis of camera spectral sensitivities, and proposed op- 
timization approach; Section 4 demonstrates a number of 
experiments on synthetic and real data. This is followed by 
a discussion in Section 5. 


2. Related Work 


Most commercial systems for HSI provide hardware that 
captures a large number of images with a tunable narrow 
band filter [12]. Multiples image are taken with a spec- 
tral filter that only allows spectral energy at a certain wave- 
length to pass through the filter. This process is repeated 
for a set discrete of wavelengths. A HSI system that pro- 
vides 31 bands (every 10nm between 400nm and 700nm) 
would need to take 31 images, each image with different 
spectral filter. Another commercial option is to employ a 
pushbroom imaging framework to reconstruct the spectrum 
column by column [21]. In these systems, a column of light 
enters the camera and is passed through a prism or a defrac- 
tion grid to decompose the light into its individual wave- 
lengths that is then recorded by the camera sensor. The full 
hyperspectral image is reconstructed by filling each line by 
rotating the camera. While commercial hyperspectral cam- 
eras provide accurate spectral measurements, the hardware 
requires careful control of mechanical components that sig- 
nificantly increase the cost of the equipment. Another prob- 
lem is that the image resolutions for these systems are often 
low compared to conventional cameras, so super-resolution 
algorithms may be necessary to increase the resolution as 
described in [17]. 

There have been a number of works that propose alter- 
natives to tunable filters or push-broom designs. For ex- 
ample, the work in [31] reconstructed a multispectral video 
from RGB images by capturing a scene under a set of light 
sources with different spectral power distributions. The key 
component of their system is a technique to determine the 
optimal multiplexing sequence of spectral sources in order 
to minimize the number of required images for HSI. The 
work in [6] also took advantage of active lighting by us- 
ing an optimized wide band illumination to obtain multi- 
spectral reflectance information. Instead of putting the 
spectral filters in front of the camera itself, the key idea of 
the work in [6] is to put the spectral filters in front of the 
illumination. While these active illumination methods pro- 
vide an effective means for HSI, they do require expertise 
to build and use. Another major limitation is that they can 
only be used indoors under controlled lighting conditions. 

Instead of using active illuminations, fast algorithms for 
multispectral video capture were proposed by using a prism 
in [11] and a DLP projector in [13]. In [11], a prism 
was used to separate the incoming light’s spectra. An op- 
tical mask was placed in front to avoid overlap between 
neighboring rays that would make the boundaries between 
the different pixel’s spectra ambiguous. An unique color- 
forming mechanism via DLP projectors combined with a 
high speed camera was exploited for spectral reflectance re- 
covery in [13]. A common difficulty in using these systems 
is expertise necessary to set up the required hardware sys- 
tems. 
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Single image multispectral imaging algorithms have also 
been proposed. Since an RGB camera provides three mea- 
surements per pixel only, it is an ill-posed problem to re- 
cover the higher dimensional signal per pixel directly from 
a single image. Single image methods therefore need to im- 
pose strong assumptions on the surface reflectance and rely 
extensively on associated training data to constrain the so- 
lution. To model the mapping from an RGB signal to higher 
dimensional spectral signal, prior single image methods 
have performed reconstruction using a metamer-set [26], or 
reconstruction using linear [1] and non-linear [28] interpo- 
lation using the associated training data. The results of these 
methods depend highly on the training data and their simi- 
larity to the imaged scene. 

Compared to the aforementioned methods, the HSI 
method proposed in this paper offers several advantages. 
First, we only require the use of multiple commodity cam- 
eras; special filters, lights, etc., are not required. This makes 
the system relatively low-cost and easy to use. Our ap- 
proach is also able to recover hyperspectral images much 
more accurately as compared to single image based meth- 
ods. In addition, by using commodity cameras, our method 
inherently provides high resolution hyperspectral images. 
Since we simultaneously recover both the surface spectra 
and the illumination spectra, an extra stage for light separa- 
tion as performed in [18] is unnecessary. Lastly, our system 
can be used both indoors and outdoors. 


3. HSI Algorithm 
3.1. Problem Formation 


We first introduce the imaging model of digital RGB 
cameras. We assume Lambertian surface with a uniform 
illumination for the whole scene, and also assume that im- 
ages for different cameras were taken under the same light- 
ing condition. Another important assumption for this work 
is that the spectral sensitivities (or camera responses) for the 
cameras are known. A pixel intensity of an image from m"” 
camera can be expressed as: 


Dm,k(@) = J s021Q)ene Ad. (2) 


Q 


where pyn,x() is the intensity of a pixel x in the k*” channel 
of the image from the m‘” camera, Q is the range of the 
visible spectrum, s(A, 2) is the spectral reflectance of the 
scene point x, /(A) is the spectral power distribution of the 
illumination, and c,,, (A) is the spectral sensitivity of m-th 
camera for the k*” channel. 

It is widely known that surface spectral reflectance of 
real-world materials can be well approximated using a lin- 
ear combination of a small number of spectral basis [7, 25, 
32]: 


s(A, x2) = ri (x)bi(A), (3) 
i=1 

where N,. is the number of the reflectance basis, b;(A) is 
the basis function of the spectral reflectance, and r;(x) is 
the corresponding coefficient for the i*” basis. In this work, 
we compute the basis functions b;() by running Principal 
Component Analysis (PCA) on the dataset that contains the 
measurement of spectral reflectance of 1257 Munsell color 
chips [32]. The number of basis was set to 8 (i.e. N, = 8), 
which is able to explain more than 99% of the total variance 
of the data. 

We model the illumination /(X) in a similar fashion as 
the spectral power distributions of real-world illumination 
is also known to lie in a low dimensional space [16, 33]. 
This can be expressed as: 


Na 
U(r) = S> aje;(A), (4) 
j=l 


where N, is the number of illuminant basis, e; (A) is a basis 
function for illuminant spectra, and a; is the correspond- 
ing coefficient. To compute the basis functions, we use the 
database from [3] which contains spectra of 102 illumina- 
tions. We perform PCA separately on the outdoor and in- 
door illuminants. We use 65 illuminants for outdoor scenes 
and use all 102 illumination for indoor scenes. The number 
of basis, Ng, is set to 4 for outdoors, and 6 for indoors. 

Combining our models for surface reflectance and scene 
illumination, we can rewrite Eq. 2 to obtain: 


Nr Na 


pede pcaives / Oe end 
i=1 j=l a 


(5) 


ri(2)aj;Am,b(i, 7); 


where Ajnx (i, 7) = ff bi(A)ej(A)em,n (A) dA. 
The above equation can be expressed in a matrix format 


as: 

Pm,k(2) = 1(2)" Ama, (6) 
where r(x) = [ri(x),ro(x), ---,rn,(x)|7, a = [a, 
a2,+++ ,an,]7, and Ayn, isa N, x Ng matrix. 


For an image with n pixels, the intensity and the surface 
reflectance at every pixel can be rearranged to obtain: 


Pm,k = RT An 4a, (7) 


where Pmt = — [Pm,k(1),Pm,e(2),°°*  Pma(n)]” is 
the pixel intensity vector of length n, and R = 
[r(1),r(2),--- ,r(m)] is the N,. x n surface reflectance ma- 
trix. 
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Figure 2. The percentage of the variance with growing number of 
basis for R,G,B channels separately. We can observe that the space 
is close to being 8D. 


This bilinear system in Eq. 7 is the final formulation that 
forms the core of our spectral imaging system. The goal 
now is to compute both the surface reflectance R and the 
illumination spectrum a from multiple observations of the 
scene from different cameras. Using NV. number of cameras 
gives us N,. x 3 observations as each camera provides three 
color channels. It is important to note that the intensity val- 
ues from cameras must be from camera RAW images as the 
values from regular JPEG images are heavily processed vio- 
lating our imaging model [20]. We used the dcraw software 
to obtain linear RGB images from camera RAW data. 


3.2. Analysis of the Spectral Sensitivities of Cam- 
eras 


The premise of our work is that different cameras pro- 
vide different samples of the spectrum to enable the full re- 
construction of the spectrum when combined. This means 
the accuracy of the estimated hyperspectral signals obtained 
by solving Eq. 7 depends on the relationship between the 
spectral sensitivities of different cameras. The best scenario 
would arise when the spectral responses are narrow band 
in nature with no overlap between different cameras. The 
worst case would be when the spectral sensitivities of dif- 
ferent camera models are almost identical. 

We analyzed the spectral sensitivities of different cam- 
eras as done in [15, | 8] to validate that they provide enough 
independent measurements of the incoming light spectrum. 
The space of the camera spectral response for each chan- 
nel was reported to lie in two dimensional manifold in [15] 
and a three dimensional manifold in [18]. We combined the 
data provided in [15, 18] and performed PCA on a dataset 
of 40 cameras. The percentage of the variance with grow- 
ing number of basis is plotted in Fig. 2. While the space 
of the spectral sensitivities is low, we can observe that they 
are close to being eight dimensional for all three channels 
together (e.g. two basis for the green, three basis for the 
red and the blue channels respectively). This eight dimen- 
sional basis provides enough variance to solve our problem 
in Eq. 7. 


3.3. Optimization using Alternating Least Squares 


The bilinear system in Eq. 7 can be solved by minimiz- 
ing the following objective function in a least squares sense 
with respect to R and a: 


N. 3 
R,a=argmin )> 7 [Pmk-R7Amxald- 8) 


™ m=1k=1 


However, there are additional constraints we can place on 
the solution as follows: 


Ne 3 
R, 4 =arg re X » [Pm,k — R? Ay, pals 
n 68(A, z) 4 
ray f (S37) dy 


(9) 


r=19 


“fa 


st. s(A,x), (A) > 0 forall A, 2. 


In Eq. 9, we imposed an additional positivity constraints 
as both the surface and the illumination spectra should be 
positive. We also impose a smoothness constraint on both 
the surfaces and the illumination as this is often observed in 
real world surfaces and illumination spectra. 

The objective function can be expressed in matrix form 
as follows: 


N 3 


3 > |Dm,k _ R? A, pal? 


R,a = ore | 
R,a 
m=1k=1 
(10) 
+ a||WBR)|j. + 5|WEa\; ¢, 


st. BR,Ea> 0, 


where W is the second-order difference matrix, B,; = 
b;(v) with i is from 1 to N,, E,,; = a;(v) with 7 is from 
1 to Ng, and v is from 1 to 31. v represents 31 bands from 
400nm to 700nm with the intervals of 10nm. 

A least squares solution for this system of bilinear equa- 
tions can be found by iteratively solving the two linear sub- 
problems [2, 8]. To minimize Eq. 10, we adopt the alternat- 
ing least squares method in [2] and alternate between solv- 
ing for the illumination a by fixing the surface reflectance 
R and then solving for R with fixed a. We have empirically 
found that the initialization of R does not significantly af- 
fect the results, and we initialize every spectral reflectance 
as the first reflectance basis. Details on the alternating least 
squares optimization steps are included in the supplemen- 
tary material. 
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Figure 3. The spectral sensitivity of the 6 cameras used in this work (Canon 5D, Fujifilm X-E1, Leica X2, Nikon D600, Nikon D800, and 


Sony Nex-3N). 


3.4. Hyperspectral Image Reconstruction 


Our HSI framework relies on multiple observations from 
different cameras. To reconstruct a high resolution hyper- 
spectral image, a registration process is necessary to find 
correspondences between images to build the observations 
vectors Py,~ in Eq. 7. To allow us to focus on the HSI 
reconstruction, we captured planar scenes so that homogra- 
phies could be used for the registration. Note that our hyper- 
spectral imaging algorithm itself is general that can be used 
for non-planar scenes with a dense registration method such 
as dense stereo matching [34], patch-match [4], etc. While 
such registration is possible, it comes with its own chal- 
lenges that is outside the scope of this paper. 

Solving the bilinear system iteratively for all the pixels 
in the high resolution images can be extremely slow as the 
size of matrix R grows with the number of pixels. For fast 
computation, we first solve Eq. 10 for a selected number of 
points (e.g. 30 points only). Since the solution provides the 
illumination spectrum a, the rest of the surface reflectance 
spectra R can now be solved linearly with the known illu- 
mination spectrum a. 


4. Experiments 
4.1. Camera Spectral Sensitivity 


Before running our algorithm to reconstruct hyperspec- 
tral images, we need to compute the spectral sensitivities of 
the cameras employed in our system. We used six cameras 
in this work as follows: Canon 5D, Fujifilm X-E1, Leica 
X2, Nikon D600, Nikon D800, and Sony Nex-3N. To esti- 
mate the spectral sensitivities, we measured the spectral sig- 
nals of the Macbeth color chart and additional color chips 
using a hyperspectral camera (Specim PFD-V10E) and cap- 
tured images of the same object with our 6 cameras at the 
same time. We perform PCA on the camera spectral sen- 
sitivity database from [15] and [18], and the first five basis 
functions were used for the estimation of the sensitivities. 
Spectral sensitivity ¢;, was computed by solving the follow- 


Reconstruction error 


1 2 3 4 5 6 
Number of cameras 
Figure 4. The average error for estimated reflectance and illumi- 
nation according to the number of cameras used. 


ing equation: 
Ce = Hed, (1) 


where 
qx =arg min |p, — DHiqx|3, 
Gk (12) 
8.t. Hyde = 0, 


where c;, is camera spectral sensitivity of the k” channel, 
Hy, is the matrix of the basis functions, qy is the corre- 
sponding coefficient vector, px is the observed pixel inten- 
sity of color chart in channel k, and D is the matrix of 
stacked color chart patches’ hyperspectral signal. Fig. 3 
shows the spectral sensitivities of our 6 cameras. 


4.2. Experiments on Synthetic Data 


We first perform experiments on synthetic data to vali- 
date our algorithm as well as to analyze the effect of the 
number and the selection of the cameras. Ground truth 
spectral reflectance of the Macbeth color chart patches and 
samples of illumination spectra from [3] were used to gen- 
erate hyperspectral scenes of the Macbeth Chart. We then 
generated RGB values of each color patch for each cam- 
era according to Eq. 2 with the computed camera spectral 
sensitivities . While generating the RGB values, we added 
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(k) Recovered spectral reflectance of 24 patches from Macbeth color chart 


Figure 5. Outdoor experiment under sunlight using a Macbeth color chart. The results from our method and the single image based 
method [28] are compared. (a)-(c) RGB inputs from three different cameras, (d)-(f) estimated illumination spectrums and pixel-wise 
surface reflectance reconstruction error of Nguyen ef al.’s method [28] and our method, (k) recovered surface spectral reflectance of 24 


color patches using the two methods. 


zero mean Gaussian noise with a standard deviation of 15 to 
simulate noise as well as mis-registration between images. 

Fig. 4 shows the root-mean-square-error (RMSE) for 
both the reconstructed color patches and the reconstructed 
illumination as the number of used cameras increases. To 
investigate the effect of the number of cameras, we test all 
63 combinations of our cameras. The results in Fig. 4 report 
the average errors per number of cameras. As expected, us- 
ing a single camera image is not sufficient, however, the 
reconstruction error begins to stabilize starting with two 
and more cameras (especially for the surface). The anal- 
ysis shows that using three cameras would be good for both 
the effectiveness and the accuracy, and it also shows that us- 
ing more than four cameras is not necessary as it does not 
increase the accuracy. This is expected as the space of the 
camera spectral sensitivity is low and this analysis also fits 
with our observation in Fig. 2. 

With the above analysis, we fixed the number of cameras 
to be used as three. The criteria for finding the best com- 
bination of camera should depend on linear independence 
of the cameras spectral sensitivities. For camera selection, 
we build a matrix with columns being the spectral sensitiv- 


ities and chose the combination with the lowest condition 
number (the ratio between the maximum and the minimum 
singular values of the matrix). The combination of Canon 
5D, Fuji X-E1, Nikon D600 cameras produced the lowest 
condition number and indeed resulted in the most accurate 
reconstruction. The following examples of real scenes were 
all conducted with the above combination. 


4.3. Experiments on Real Data 


For the experiments on real data, we captured hyper- 
spectral images of several planar objects under different 
illumination with a hyperspectral camera as ground truth 
data. The same scenes are photographed using the selected 
three cameras. For the real data experiments, homographies 
from each image to the ground truth hyperspectral image 
are computed for the registration and the resolution of the 
examples in this paper is 200 x 300 that covers a single 
planar object. With a Matlab implementation, it takes 2 
minutes for our algorithm to reconstruct the hyperspectral 
image with that resolution. 

Fig.5 compares HSI results of our method and the sin- 
gle image based method in [28]. For this experiment, a 
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Figure 6. Hyperspectral imaging results of various planar scenes under indoor and outdoor illuminations. Results of our method and 
Nguyen et al.’s method [28] are compared. The ground truth spectral reflectance is rendered as RGB images using spectral sensitivies of a 
Canon 5D camera for reference. The errors indicate RMSE in the hyperspectral domain. 
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Figure 7. Recovered reflectance of paintings using the proposed method. RGB images of paintings taken under both indoor and outdoor 
lighting conditions are converted to hyperspectral images. The ground truth and the reconstructed hyperspectral images are rendered as 
RGB images using the spectral sensitivies of the Canon EOS-5D camera. The errors indicate RMSE in the hyperspectral domain. 


Macbeth color chart was imaged outdoors under the sun- 
light. As can be seen, the hyperspectral signals of both the 
illumination and the patches recovered by our method are 
more accurate than the ones recovered by [28]. In addition 
to Fig. 5, results on more various scenes under both indoor 
and outdoor illuminations are shown in Fig. 6. The color in 


the error maps indicates pixel-wise RMSE in the hyperspec- 
tral domain. The proposed method constantly outperforms 
the previous method that uses single RGB image [28] on 
various scenes and lighting conditions. 


Fig. 7 shows reconstructed hyperspectral images of 
paintings under both indoor and outdoor illuminations. 
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Figure 8. Relighting applications. Spectral surface reflectance estimated by our method are relit using 4 different illuminations and rendered 
as RGB images using the camera sensitivity of a Nikon D600 camera. The relighting results are compared with ground truth images: scenes 


taken under the specified illumination. 


Both the ground truth and the reconstructed hyperspectral 
images are visualized as RGB color images. The recon- 
struction is satisfactory except for some edge regions, where 
the registration may not be accurate enough. The estimated 
spectral reflectance can be used to relight the scene. Fig. 8 
illustrates the use of our method on relighting. 


5. Discussion 


We have introduced a framework for affordable and 
easy-to-use hyperspectral imaging using commercial digi- 
tal cameras. Our system exploits the variation in the spec- 
tral sensitivities of cameras and combines the different color 
measurements into a full hyperspectral signal. We have 
demonstrated the effectiveness and the accuracy of our sys- 
tem using various examples both indoors and outdoors, and 
demonstrated the ability to use the hyperspectral data for 
tasks such as scene re-lighting. 

While our system was able to reliably reconstruct hyper- 
spectral images in most cases, it ran into a problem when 
the system was used under a fluorescent illumination. This 
is due to the rather peculiar spectrum of fluorescent lights 
as shown in the fourth example of Fig. 8. The energy is 
focused on a sparse set of spectral bands for this type of 
lighting and the variation of observations by different sen- 
sors become negligible for our system to work well. 


There are several interesting directions for future work. 
From a practical viewpoint, we would like to extend the 
work to work with 3D scenes by applying dense stereo 
matching registration. With our framework and a 3D re- 
construction system, we could generate 3D hyperspectral 
texture-maps. Additionally, we would like to remove the 
need to know the spectral sensitivities of the camera by in- 
corporating this into the framework. This would mean that 
we could estimate everything at once, including surface re- 
flectance spectra, light spectra, and camera spectral sensi- 
tivities all from only camera input images. 
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